Passive or active monitoring devices can perform live network traffic inspection to detect a network attack. Offline analysis can be performed on packets determined to be involved in the attack that were captured and stored to long-term storage, such as to a disk. However, detection of the network attack may rely on detection of signatures and/or behaviors characteristic of attacks that may not be detected until a significant portion of a traffic stream including the determined packets has already been inspected.
Conventional methods and systems of monitoring traffic streams include capture and store to long-term storage all packets in the traffic streams being monitored in order to analyze the stored packets in the event that an attack is detected. Thus, packets included in traffic streams that are unrelated to an attack in addition to packets in traffic streams that are related to such attacks are stored in the long-term storage, requiring large quantities of long-term storage. The long-term storage used for capturing network traffic typically uses high-cost components that have the capability to operate quickly enough to keep-up with speed of the network traffic. In addition to high costs associated with the long-term storage, management of the large quantities of archived packets is complex and requires a high degree of overhead. Data retention time is limited due to the large quantities of data being stored. Additionally, slow data mining operations performed on the large amounts of stored data degrades user-experience.
Such conventional methods and systems have generally been considered satisfactory for their intended purpose. However, there is still a need in the art for reducing the amount of data stored for off-line analysis. The present disclosure provides a solution for these problems.